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ABSTRACT 

Research and development of the Advanced Display and 
Computer-Augmented Control System (ADCACS) for the 
space station Body-Ported Cupola Virtual Workstation 
(BP/VCWS) have been pursued at the Marquette Universi- 
ty Advanced Control Technology (ACT) Laboratory. This 
project explores the potential applications of body-ported 
virtual display and intelligent control technology for the 
human-system interfacing applications in space station 
environment. The new system is designed to enable crew 
members to control and monitor a variety of space 
operations with greater flexibility and efficiency than 
existing fixed consoles. 

The technologies being investigated include helmet- 
mounted virtual displays, voice and special command 
input devices, and microprocessor-based intelligent 
controllers. Several research topics, such as human 
factors, decision support expert systems, and wide field 
of view, color displays are being addressed. A prototype 
integrated emulator is in development. Our study has 
demonstrated the significant advantages of this uniquely 
integrated display and control system, and its feasibility 
for human-system interfacing applications in the space 
station command and control environment. 


1. Introduction 

The NASA Space Station Freedom Cupola workstations 
will play a key role in supporting crew to control and 
monitor many critical operations and systems in their 
missions. Examples include station manipulators, flight 
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telerobotic servicer, station mobile transporter, control of 
external video and lights, etc. To support these functions, 
the cupola workstation must support direct or video 
observations of operations of payloads, docking and 
berthing, remote inspection of any external point on space 
station structure, EVA operations, and other operations. 
Clearly, the performance of display and human-system 
interface technology in the space station cupola environ- 
ment directly affect the efficiency and productivity of the 
space mission, and the crew safety. The cupola 
workstation must provide a human-system interface for 
task control and monitoring which accommodates both 
direct viewing through any of the cupola windows, and 
remote viewing through any of the space station’s video 
cameras, including those on telerobots, on remotely 
controllable vehicles, and on the station itself. 

The multi-directional viewing requirements indicate that 
there is a potential for conflict between the spatial orienta- 
tion of a fixed, or "hard console" cupola workstation and 
the directions of external tasks. A fixed workstation favors 
a single direction, while disfavoring other directions. In 
addition, the need to divide attention between external 
views and panel mounted displays will create problems 
for the crew operator. Further disadvantages of perma- 
nently installed workstations in the cupola include a 
potential conflict with the requirement that cupola windows 
must be replaceable on orbit, since the fixed displays are 
likely to abut, or even block cupola windows. Currently 
specified workstations will require multiple displays for 
each workstation, with a penalty of weight, volume, and 
power. 

The fast growing virtual reality and head-mounted display 
technologies offer an attractive alternative to the fixed 
console displays. They are portable, easy to install and 
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easy to maintain. In addition, head-mounted display 
devices and their control processors are compact, light- 
weight, and low power. This new technology enables the 
cupola workstation to offer multiple displays, whose 
visibility depends on head position and orientation 
tracking. Moreover, the possible applications of state-of- 
the-art microcomputer hardware and intelligent control 
technologies in the cupola workstation design open a 
promising approach which allows the keyboard, control 
input devices, and audio input/output devices to be body- 
ported by the crew operator. With all the new ideas 
described above, a new design concept, called "Body- 
Ported, Virtual Cupola Workstation (BP/VCWS)' 1 , can be 
formulated. 

Use of head-mounted virtual display and intelligent control 
technologies in the cupola workstation design, however, 
create new problems that should be solved before 
physical construction of the BP/VCWS. First, this newly 
proposed cupola workstation design involves advanced 
technologies in multi-disciplines, including a helmet- 
mounted virtual display device, voice and special com- 
mand input devices, and microprocessor-based intelligent 
controllers. Several research topics, such as human 
factors, decision support expert systems, and wide field 
of view, color displays must also be addressed. More- 
over, the effectiveness and user acceptance of such a 
device is another key issue. 

Most head-mounted display technologies, which are 
currently available for application to the space station has 
been in use for aviation. Representative systems devel- 
oped for use in combat rotorcraft or fighter aircraft include 
the Honeywell Integrated Helmet and display Sight 
System (HADSS). Two systems developed for use as 
visual systems for flight simulators are the Visually 
Coupled Airborne Systems Simulator (VC ASS), developed 
at Wright-Patterson AFB, OH, and the Fiber-Optic Helmet 
Mounted Display (FOHMD), developed at Williams AFB, 
AZ. All of these systems have been developed for 
specific application environments which are very different 
from that of the space station. For example, there is no 
need of a crash helmet during missions on board the 
space station, so the integration of a display into a helmet 
is of little use. The tradeoffs required for such integration 
are inappropriate for application to space station. The 
simulator displays, on the other hand, are very bulky and 
would be unacceptable as flight equipment. 

NASA has two head-mounted display activities that are 
relevant to the current Advanced Development effort. One 
project is the development at JSC of prototype displays 
for use inside EVA helmets. The EVA helmet display has 
considerable power constraints which limit the design 
options. Further, the information to be presented on the 


EVA display is of far less complexity than the potential IVA 
display. Overall, the design goals of the IVA system 
would not be met by the EVA system. The other NASA 
project is the development at ARC of low-cost, wide field- 
of-view head-mounted displays for laboratory research in 
Virtual Environment Display (VIVED). This system does 
not provide the see-through display capability which will 
be required on the space station. Further, the ARC 
system provides a very wide instantaneous visual field-of- 
view at the expense of resolution, which is the opposite of 
the design tradeoff needed for the space station display. 

The research and development effort for such a BP/VCWS 
system has already been pursued at the Marquette 
University Advanced Control Technology (ACT) Laborato- 
ry. The research project, "Advanced Display and Comput- 
er-Augmented Control System (ADCACS) 11 , is sponsored 
by NASA Ames research center and includes the partici- 
pation of Astronautics Corporation of America. The 
project motivates the exploration of the potential applica- 
tions for the virtual display and intelligent control technolo- 
gies especially in the space station environment. The new 
system will be designed to enable crew members to 
control and monitor a variety of space operations with 
greater flexibility and efficiency than existing fixed con- 
soles. 

The technologies being investigated include helmet- 
mounted virtual displays, voice and special command 
input devices, and microprocessor-based intelligent 
controllers. Several research topics, such as human 
factors, decision support expert systems, and wide field 
of view, color displays are being addressed. A prototype 
integrated emulator is in development. Our study has 
demonstrated the significant advantages of this uniquely 
integrated display and control system, and its feasibility 
for human-system interfacing applications in the space 
station command and control environment. 


2. ADCACS Project Overview 

The Advanced Display and Computer Augmented Control 
System (ADCACS) was proposed as a three-year systems 
engineering project that is devoted to the research and 
development of such an advanced human-system 
interface device, i.e., the Body-Ported Virtual Cupola 
Workstation (BP/VCWS). This project takes advantage of 
other closely related on-going projects that are critical and 
very important to the success of the project. 

Objectives 

Specifically, the overall objectives of the ADCACS project 
are: 
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1) to demonstrate the feasibility of the Body-Ported 
Cupola Workstations (BP/VCWS) meeting the 
needs of the baseline and evolutionary Space 
Station Freedom (SSF) Cupola Workstations; 

2) to initiate an engineering approach to construct 
such a BP/VCWS, and through evolution, to re- 
place existing "hard console" cupola workstation 
design; 

3) to develop a flight-qualifiable prototype BP/VCWS, 
and to significantly advance the general state-of- 
the-art in workstation technology including virtual 
reality" and intelligent Control. 

Evolutionary Approach 

The above goals were planned to be accomplished in the 
three-year project that was scheduled in three phases. 
The Phase one research, started July 1 , 1 990 and ended 
June 30, 1991, has been carried out at the Marquette 
University Advanced Control Technology (ACT) Laborato- 
ry, with participation of Astronautics Corporation of 
America (ACA). During the Phase One research, it was 
necessary and important to establish an evolutionary 
approach toward the physical construction of the flight- 
qualifiable BP/VCWS, starting from the requirements and 
feasibility analysis to the construction and evolution of on- 
ground emulators. A variety of factors were taken into 
account. 

The evolutionary steps can be described as: 

1) to define requirements and configurations for the 
BP/VCWS that are consistent with crew user 
requirements and Space Station Program con- 
straints; 

2) to study the relevant technologies, that will be 
applied to the proposed body-ported virtual display 
and knowledge-based intelligent control system, 
through literature search and laboratory experimen- 
tal investigations; and 

3) to build the prototype emulator, and later build the 
Full-Function Emulator (FFE), based on the proto- 
type emulator for further studies of technology 
readiness of flight-qualifiable BP/VCWS. 

3. Description of the Prototype Emulator System 

3.1 Overview of the Virtual Workstation Emulator 

A prototype emulator for the virtual display and intelligent 
system has been designed and built during the first year. 


This emulator environment consists of an IBM-386 
compatible personal computer (PCI ), an IBM-286 compat- 
ible personal computer (PC2), and a Silicon Graphics 
Personal IRIS workstation (IRIS). The IRIS, model 
4D/35TG has a 35MHZ CPU. This architecture is shown 
in Figure 1 . 



Ethernet (NFS/RPC) 


Figure 1 Overview of the Prototype Emulator Environ- 
ment 

As shown in Figure 1, the PCI is dedicated to the intelli- 
gent control and voice command input experiments, and 
PC2 is used for acquiring and processing data from the 
ftngers/hand/arm position sensing input devices. The IRIS 
is used for the graphic generation of the virtual scenario 
elements to be manipulated. The entire system is 
networked together using an Ethernet LAN. This proto- 
type emulator can perform the following experiments: 

1) virtual tele-robot arm/hands display and control; 

2) voice command input; 

3) force-based arm and position-based master 
command input and control; 

4) knowledge-base intelligent control and decision 
support system testing and implementation. 


3.2 Networking 

For inter-process communication over the Ethernet LAN, 
the "Sockets" protocol was chosen for the ADCACS 
application. RPC (remote procedure call) and NFS 
(network file system) were also evaluated as options to 

sockets. 

Internet Domain Stream Sockets was chosen for its 
simplicity in satisfying the ADCACS project needs. With 
proper programming, all subsystems can process data 
concurrently. Any subsystem may be a client or a server. 
This increases the flexibility of the entire emulator. Minimal 
overhead is required since sockets is a fairly low level 
communications technique. 
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3.3 Graphics Generation 

Graphical generations for virtual environment and virtual 
tele-robot are performed on the Cilison Graphics IRIS 
workstaiton. This machine is fast enough to generate the 
real-time (no perceivable delay) images necessary for the 
present research. It is anticipated that as the complexity 
of the control scenarios increases, the graphical image 
generation power will have to be increased. 

The primary graphics program in use at the present time 
is "armbox.c". It uses the Silicon Graphics standard 
graphics library, GL. GL is a set of C callable subrou- 
tines. The subset of routines being used in "armbox.c" 
includes: 1) a variety of viewing windows; 2) a single 
infinite light source (multiple infinite and local lighting 
source models are available for experimentation and 
require increased processing time); 3) RGB mode and 
Gouraud shading for realistic shading with fewer polygons 
as compared with flat shading; 4) double buffering to 
synchronize rendering with the screen so smooth motion 
exists with no flickering; 5) backface removal and z- 
buffering to decrease rendering time for hidden surface 
removal; 6) viewing matrix manipulation and stack routines 
to easily and quickly position graphical elements; 7) 
queues for internal events and standard input devices; 
and 8) simple 3 dimensional polygon rendering routines. 

The program routine "armbox.c" renders a picture of a 
right arm, including a hand and fingers using positioning 
data received over the network. It also has techniques for 
drawing boxes in any location and orientation. For 
iteration experimentation of the graphics portion of the 
emulator, a second process may be run on the Personal 
IRIS which emulates a microcomputer sending data to the 
IRIS. 

For the networking setup, the IRIS is the client and PC2 
is the server. The main graphic loop requests positioning 
data from the server. It then immediately renders a 
picture using initial data. After rendering the picture, it 
reads the data sent from the server and loops. The 
second time through the loop it again requests data from 
the server and immediately renders a picture, this time 
with the previous data. Each time though the loop the 
picture is rendered using the previous data. This tech- 
nique allows the graphic generation to be performed in 
parallel with the acquiring and preprocessing of the 
positioning data. 

The screen is refreshed at 60 Hz so there is no perceiv- 
able flicker. The image update time appears to be slower 
than 60 Hz. Until the next image is generated, the IRIS 
simply re-displays the same image. For the same 3 


dimensional model, the image generation time will differ 
depending on the depicted position of the virtual arm. 

Collision detection has been successfully completed for 
boxes in any orientation. In addition to drawing an arm, 
several boxes are also drawn within arm’s reach. The 
location of the tip of each finger is compared to the 
location of a box resulting in a Boolean array of collision 
locations. Additional points on the fingers, hand, or arm 
may be identified as important for collision detection and 
added to the Boolean array. This may be done for any 
number of boxes. 

Once the collision detection data is acquired, decisions 
will be based on the data. These decisions could include 
coloring the appropriate box, limiting motion of the 
moving object (a finger presently), or grabbing and 
moving one of the boxes. If these decisions are simple 
and need to be made quickly, they could be performed 
internal to the IRIS. If they are complex and could be 
made with a longer time delay, it may be appropriate to 
off-load the decision process to another processor on the 
network to preserve processing time on the IRIS for 
complex graphical generation. The decision processor 
could be running a knowledge-based software package. 

The present collision information is used to draw the 
appropriate box with a color that is dependent on which 
finger is touching the box. This gives the operator clear 
visual feedback. 

Also, a virtual space cupola environment model has been 
developed on the IRIS for environmental and space 
operation monitoring purposes. This model is connected 
to the knowledge-based system on PCI through Ethernet 
Link. 

3.4 Master Command Input Devices 
Force-Based Master Arm 

A force-based master arm controls the prototype emulator 
environment. The arm controller utilizes 5 degrees-of- 
freedom on the arm (i.e. shoulder flexion/extension, 
shoulder abduction/adduction, upper arm rotation, elbow 
flexion/extension, and forearm rotation) with near future 
goals of wrist flexion/extension and wrist 
abduction/adduction. The current configuration consists 
of the force-based arm as an input device which sends an 
analog signal to an A/D board and the interpretation 
software in a 286-based microcomputer (PC@). This 
computer reads a digital signal from the 5 degrees-of- 
freedom on the arm and maps it to a range of motion for 
the virtual arm. This information is then sent across a 
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distributed network to the graphics workstation via 
Internet Domain Stream Sockets. This data is read by the 
graphics terminal and is used to adjust the position of the 
virtual human arm that is graphically portrayed. 

The interpretation that occurs at PC2 consists of scaling 
the digital output of each channel to a range of motion. 
This is accomplished through the utilization of a physio- 
logical data base called Ergobase by Biomechanics 
Corporation of America. This data base contains the 
range of motion for each degree-of-freedom that is 
addressed by the force-based arm. Ergobase contains 
the ranges of motion in degrees for the above specified 
movements for the central 95 percentile of the population. 
By linearly transforming the digital output of the force- 
based sensor to an angle degree that is within the human 
range of motion for the specific joint, a truly anthropomor- 
phic graphical arm in the virtual world results. 

The prototype emulator has been created with optimum 
anthropomorphic and human factor considerations at 
every step of its design. The design of virtual scenario 
presentation will be dependent on calculations and 
analysis of optimal human range of motion. 

Position-Based Master (PBM) Hand 

The position based master (PBM) hand has been suc- 
cessfully added to the fingers/hand/arm position data 
processing microcomputer to complement the force- 
based arm. Each joint of the PBM hand has an associ- 
ated potentiometer (pot). The resulting voltage signal 
from each pot (some limited range within 0 to 5 volts) is 
sent through a two stage operational amplifier circuit. The 
first stage isolates the signal from the rest of the circuit. 
This is useful since it essentially eliminates any effects of 
unmatched pot and wire resistances in the PBM. The 
second stage has two adjustable pots. One is for gain 
and the other is for level shifting so that the resulting 
voltage ranges from 0 to 5 volts. The output of the 
second stage is properly conditioned for the full range of 
the analog-to-digital converter. 

3.5 Knowledge-Based Intelligent Control 

One 386-based personal computer is dedicated for 
experimental investigations of knowledge-based intelligent 
display system control and the decision support systems. 
A microcomputer-based speech recognition system, Voice 
Master Key produced by Convox, Inc., was successfully 
installed at the beginning of the ADCACS project. A 386 
microcomputer is designated for intelligent expert system 
control to the speech recognition system. Based on PCI 
and communication facilities through L^N, the expert 


system shell, called "Nexpert Object', is used for the 
following two intelligent control experiments: 

1) A Prototype KBS Control System (see Figure 2 for 
details), and 
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Figure 2b: Control Signal Diagram 



Figure 2c: Display Synchronization Diagram 


2) A simple control system that is based on a 
time-sequential control strategy, which uses several 
procedures implemented by one process in PCI. 
The calling out function of Expert will be used, 
which refers to the ability of an expert system 
application to call user written functions or library 
routings. The knowledge-based Expert System 
Shell was used to infer the production rules in the 
knowledge base for data validity checking and 
intelligent controlling. Also the data transform and 
inter-procedure communications were implemented 
under the Knowledge-Based System. 
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3.6 Voice Command Input Devices 
The Voice Master Key (VMKey) 

VMKey (Version 2.0) is a voice control device made by 
CONVOX INC., which can be used to train and control 
computers to accept commands in simple spoken English 
or any other languages. VMKey comes in two parts: the 
hardware "ear 11 and the software "brain". The "ear" is a 
combination headset earphone/microphone plus a plus-in 
card. This device allows the user to run an application 
program with voice commands. 

Voice Command Process Control 

The following voice command input functions have been 
implemented: 

1) VMKEY was used to accept the control interface 
signals; 

2) VMKey has been Integrated into the Expert Sys- 
tem; and 

3) The SPEECH Input Device has been employed to 
demonstrate the voice warning functions. 


4. Conclusions 

The design and development of the prototype emulator 
implements the first step of the evolutionary approach 
towards the construction of the proposed BP/VCWS 
system. Since this emulator is only a prototype, it has 
limited functions. Future work with the goal of improving 
the emulator includes integration of wide-field-of-view 
optical devices, implementation of stereoscopic display 
devices, and simulation of operational warning and 
caution systems. However, this prototype emulator 
establishes a solid foundation for the construction of the 
full function emulator for more intensive technical investi- 
gation of the proposed BP/VCWS. 
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